Search for: All records

Organizations rely on machine learning engineers (MLEs) to deploy models and maintain ML pipelines in production. Due to models' extensive reliance on fresh data, the operationalization of machine learning, or MLOps, requires MLEs to have proficiency in data science and engineering. When considered holistically, the job seems staggering---how do MLEs do MLOps, and what are their unaddressed challenges? To address these questions, we conducted semi-structured ethnographic interviews with 18 MLEs working on various applications, including chatbots, autonomous vehicles, and finance. We find that MLEs engage in a workflow of (i) data preparation, (ii) experimentation, (iii) evaluation throughout a multi-staged deployment, and (iv) continual monitoring and response. Throughout this workflow, MLEs collaborate extensively with data scientists, product stakeholders, and one another, supplementing routine verbal exchanges with communication tools ranging from Slack to organization-wide ticketing and reporting systems. We introduce the 3Vs of MLOps: velocity, visibility, and versioning --- three virtues of successful ML deployments that MLEs learn to balance and grow as they mature. Finally, we discuss design implications and opportunities for future work. 

Abstract Atmospheric gravity waves can play a significant role on atmospheric chemistry through temperature fluctuations. A recent modeling study introduced a method to implement subgrid‐scaleorographicgravity‐wave‐induced temperature perturbations in the Whole Atmosphere Community Climate Model (WACCM). The model with a wave‐induced temperature parameterization was able to reproduce for example, the influence of mountain wave events on atmospheric chemistry, as highlighted in previous literature. Here we extend the subgrid‐scale wave‐induced temperature parameterization to also includenon‐orographicgravity waves arising from frontal activity and convection. We explore the impact of these waves on middle atmosphere chemistry, particularly focusing on reactions that are strongly sensitive to temperature. The non‐orographic gravity waves increase the variability of chemical reaction rates, especially in the lower mesosphere. As an example, we show that this, in turn, leads to increases in the daytime ozone variability. To demonstrate another impact, we briefly investigate the role of non‐orographic gravity waves in cirrus cloud formation in this model. Consistent with findings from the previous study focusing on orographic gravity waves, non‐orographic waves also enhance homogeneous nucleation and increase cirrus clouds. The updated method used enables the global chemistry‐climate model to account for both orographic and non‐orographic gravity‐wave‐induced subgrid‐scale dynamical perturbations in a consistent manner. 

Abstract Matsuno–Gill circulations have been widely studied in tropical meteorology, but their impact on stratospheric chemistry has seldom been explicitly evaluated. This study demonstrates that, in a model nudged to reanalysis, anticyclonic Rossby wave gyres that form near the tropopause as a result of equatorially symmetric heating in the troposphere provide a dynamical mechanism to influence tropical and subtropical atmospheric chemistry during near-equinox months. The anticyclonic flow entrains extratropical air from higher latitudes into the deep tropics of both hemispheres and induces cooling in the already cold upper-troposphere/lower-stratosphere (UTLS) region. Both of these aspects of the circulation allow heterogeneous chlorine activation on sulfuric acid aerosols to proceed rapidly, primarily via the HCl + ClONO 2 reaction. Precipitation rates and heating rates from reanalysis are shown to be consistent with these heating and circulation response patterns in the months of interest. This study analyzes specified dynamics simulations from the Whole Atmosphere Community Climate Model (SD-WACCM) with and without tropical heterogeneous chemistry to demonstrate that these circulations influence substantially the distributions of, for example, NO 2 and ClO in the UTLS tropics and subtropics of both hemispheres. This provides a previously unrecognized dynamical influence on the spatial structures of atmospheric composition changes in the UTLS during near-equinox months. 

In modern Machine Learning, model training is an iterative, experimental process that can consume enormous computation resources and developer time. To aid in that process, experienced model developers log and visualize program variables during training runs. Exhaustive logging of all variables is infeasible, so developers are left to choose between slowing down training via extensive conservative logging, or letting training run fast via minimalist optimistic logging that may omit key information. As a compromise, optimistic logging can be accompanied by program checkpoints; this allows developers to add log statements post-hoc, and "replay" desired log statements from checkpoint---a process we refer to as hindsight logging. Unfortunately, hindsight logging raises tricky problems in data management and software engineering. Done poorly, hindsight logging can waste resources and generate technical debt embodied in multiple variants of training code. In this paper, we present methodologies for efficient and effective logging practices for model training, with a focus on techniques for hindsight logging. Our goal is for experienced model developers to learn and adopt these practices. To make this easier, we provide an open-source suite of tools for Fast Low-Overhead Recovery (flor) that embodies our design across three tasks: (i) efficient background logging in Python, (ii) adaptive periodic checkpointing, and (iii) an instrumentation library that codifies hindsight logging for efficient and automatic record-replay of model-training. Model developers can use each flor tool separately as they see fit, or they can use flor in hands-free mode, entrusting it to instrument their code end-to-end for efficient record-replay. Our solutions leverage techniques from physiological transaction logs and recovery in database systems. Evaluations on modern ML benchmarks demonstrate that flor can produce fast checkpointing with small user-specifiable overheads (e.g. 7%), and still provide hindsight log replay times orders of magnitude faster than restarting training from scratch. 

Abstract We use Whole Atmosphere Community Climate Model simulations made under various climate change scenarios to study the evolution of the global‐mean temperature trend in the late twentieth century and the twenty‐first century. Results are compared with available satellite observations, including new trend estimates derived from the Sounding of the Atmosphere using Broadband Emission Radiometry instrument on NASA's TIMED spacecraft. Modeled and observed trends are shown to be consistent throughout the entire middle atmosphere, from near the tropopause (~16 km) to the lower thermosphere (~95 km) in the period covered by the satellite data. Simulations are extended into the twenty‐first century to document the evolution of the global‐mean temperature trend profile. We find, consistent with previous studies, a marked change in the trend profile at the turn of the twenty‐first century, which is driven by the recovery of stratospheric ozone following the adoption of the Montreal Protocol. In the twenty‐first century, the trend profile becomes more uniform with altitude, but its overall shape and magnitude are conditioned by the scenario adopted for future emissions of greenhouse gases. Our results suggest that the vertical profile of temperature trends in the middle atmosphere will remain an important signature of global climate change, and they underscore the importance of global, continuous monitoring of this region of the atmosphere. 

Abstract The Whole Atmosphere Community Climate Model, version 4 (WACCM4), is used to investigate the influence of stratospheric conditions on the development of sudden stratospheric warmings (SSWs). To this end, targeted experiments are performed on selected modeled SSW events. Specifically, the model is reinitialized three weeks before a given SSW, relaxing the surface fluxes, winds, and temperature below 10 km to the corresponding fields from the free-running simulation. Hence, the tropospheric wave evolution is unaltered across the targeted experiments, but the stratosphere itself can evolve freely. The stratospheric zonal-mean state is then altered 21 days prior to the selected SSWs and rerun with an ensemble of different initial conditions. It is found that a given tropospheric evolution concomitant with the development of an SSW does not uniquely determine the occurrence of an event and that the stratospheric conditions are relevant to the subsequent evolution of the stratospheric flow toward an SSW, even for a fixed tropospheric evolution. It is also shown that interpreting the meridional heat flux at 100 hPa as a proxy of the tropospheric injection of wave activity into the stratosphere should be regarded with caution and that stratospheric dynamics critically influence the heat flux at that altitude.